Unable to acquire permit to use snapshot files during recovery, this recovery will recover index files from the source node. Ensure snapshot files can be used during recovery by setting [indices.recovery.max_concurrent_snapshot_file_downloads] to be no greater than [25]
this may be caused by lack of enough unfragmented virtual address space or too restrictive virtual memory limits enforced by the operating system, preventing us to map a chunk of 414627 bytes. Please review 'ulimit -v', 'ulimit -m' (both should return 'unlimited'), and 'sysctl vm.max_map_count'. More information: http://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html
OpenJDK 64-Bit Server VM warning: INFO: os::commit_memory(0x00007fe4d805c000, 65536, 1) failed; error='Not enough space' (errno=12)
# There is insufficient memory for the Java Runtime Environment to continue. # Native memory allocation (mmap) failed to map 65536 bytes for committing reserved memory. [thread 330 also had an error][thread 232 also had an error] # An error report file with more information is saved as # logs/hs_err_pid7.log
# # There is insufficient memory for the Java Runtime Environment to continue. # Native memory allocation (mmap) failed to map 65536 bytes for committing reserved memory. # Possible reasons: # The system is out of physical RAM or swap space # The process is running with CompressedOops enabled, and the Java Heap may be blocking the growth of the native heap # Possible solutions: # Reduce memory load on the system # Increase physical memory or swap space # Check if swap backing store is full # Decrease Java heap size (-Xmx/-Xms) # Decrease number of Java threads # Decrease Java thread stack sizes (-Xss) # Set larger code cache with -XX:ReservedCodeCacheSize= # JVM is running with Zero Based Compressed Oops mode in which the Java heap is # placed in the first 32GB address space. The Java Heap base address is the # maximum limit for the native heap growth. Please use -XX:HeapBaseMinAddress # to set the Java Heap base and to place the Java Heap above 32GB virtual address. # This output file may be truncated or incomplete. # # Out of Memory Error (os_linux.cpp:2728), pid=7, tid=213 # # JRE version: OpenJDK Runtime Environment Temurin-17.0.1+12 (17.0.1+12) (build 17.0.1+12) # Java VM: OpenJDK 64-Bit Server VM Temurin-17.0.1+12 (17.0.1+12, mixed mode, sharing, tiered, compressed oops, compressed class ptrs, g1 gc, linux-amd64) # Core dump will be written. Default location: /usr/share/elasticsearch/core.7 #
{ "reason" : "ALLOCATION_FAILED", "at" : "2025-01-07T16:28:22.026Z", "failed_allocation_attempts" : 5, "details" : """failed shard on node [IYowNoBkTzq_WPjXgb_10Q]: failed recovery, failure RecoveryFailedException[[notification-jiankunking-2025.01][3]: Recovery failed on {es-notify-203}{IYowNoBkTzq_WPjXgb_10Q} {nzTGfgdBQvykqKacU_cyzA}{127.0.0.203}{127.0.0.203:9301}{cdfhilmrstw}{ml.machine_memory=33568223232, xpack.installed=true, transform.node=true, ml.max_open_jobs=512, ml.max_jvm_size=17179869184}]; nested: IndexShardRecoveryException[failed to recover from gateway]; nested: EngineCreationFailureException[failed to open reader on writer]; nested: IOException[Map failed: MMapIndexInput(path="/usr/share/elasticsearch/data/nodes/0/indices/n5E4PEFpTKKDgGdNmEqSPQ/3/index/_j50_Lucene80_0.dvd") [this may be caused by lack of enough unfragmented virtual address space or too restrictive virtual memory limits enforced by the operating system, preventing us to map a chunk of 4321640 bytes. Please review 'ulimit -v', 'ulimit -m' (both should return 'unlimited'), and 'sysctl vm.max_map_count'. More information: http://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html]]; """, "last_allocation_status" : "no" }
